Papers with Adversarial testing methods
Posing Fair Generalization Tasks for Natural Language Inference (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for deep learning semantics rely on naturalistic corpora, but they often fail to support the kind of generalization we are asking for. |
| Approach: | They define and motivate a formal notion of fairness for evaluations of deep learning models for semantics . they then apply it to natural language inference by constructing challenging but provably fair artificial datasets based on the results . |
| Outcome: | The proposed evaluations show that standard neural models fail to generalize in the required ways and even these models do not solve the task perfectly. |